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Abstract: The latest advancements in augmented reality and everyday devices have led to the development of more 
compact Bluetooth and wireless technologies. This article introduces an Artificial Intelligence (AI) virtual mouse 
system that utilizes hand gestures and fingertip detection to operate computer functions through computer vision. 
The primary goal of this system is to control the computer mouse cursor and scrolling using a web camera or built- 
in computer camera as opposed to a traditional mouse. While the traditional physical mouse remains available, it 
can no longer be relied upon exclusively. While using the physical mouse there is a possibility of problems such as 
damage in the device or if it is a wireless mouse, then need a constantly keep on changing the battery of the mouse. 
This system can work very well in the Augmented Reality world, as the metaverse is constantly growing needs very 
much of proposed mouse in this metaverse. This Virtual mouse interface allows user to easily interact with 
computers by hand. 
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1. INTRODUCTION 


A virtual mouse is a software-based solution that simulates the functionality of a physical mouse on a 
computer or electronic devices. It provides user with an alternative input method. This software-based solutions 
often come bundled with accessibility featuring in operating systems, enabling user to customize the behaviour of 
virtual mouse to his specific needs and preferences. Physical mouse has proven to be invaluable for persons with 
physical difficulties or impairments. 


In this article, the proposed work uses modules such as OpenCV, mediapipe and PyautoGUI to create the 
virtual mouse system. OpenCV is used for its computer vision to help us capture the hand using the web camera of 
the computer, mediapipe is used to detect the hand using KLT Tracking Algorithm and then the K-Cosine Algorithm 
to the system is applied, K-cosine border tracking algorithm is used to identify the hand gesture and after applying 
the algorithm, PyAutoGUI is used to perform the mouse actions. 


This system will definitely help the people in making the metaverse a better place. This system can and will 
be used in the Augmented and Virtual Reality world. Image processing is a branch of computer science and 
engineering that deals with the manipulation, analysis, and interpretation of digital images. It involves using various 
algorithms and techniques to enhance, modify, or extract information from images. Image processing plays a vital 
role in numerous applications, including medical imaging, remote sensing, surveillance, and computer vision. 


One of the interesting applications of image processing is the development of a virtual mouse. A virtual 
mouse is a software-based alternative to the traditional physical mouse. It allows users to control the cursor on a 
computer screen using hand gestures or other input methods without the need for a physical mouse. 
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The concept of a virtual mouse involves using image processing techniques to track and interpret the 
movements of the user's hand or other input devices in real-time. The system captures video or image data from a 
camera or sensor, which is then processed to detect and track the position and motion of the hand or input device. 
This information is used to control the movement of the cursor on the screen, simulating the actions of a physical 
mouse. 


The image processing algorithms used in virtual mouse systems typically involve techniques such as 
background subtraction, object detection, tracking, and gesture recognition. These algorithms analyse the input data 
and extract relevant information to accurately determine the position and motion of the user's hand or input device. 


Virtual mouse technology offers several advantages over traditional physical mice. It provides a more 
intuitive and natural interaction method, allowing users to control the cursor with gestures or movements that mimic 
real-world actions. Virtual mice also have the potential to offer greater accessibility to individuals with physical 
disabilities or impairments that limit their ability to use a physical mouse. When using a wireless or Bluetooth 
mouse, some devices are used such as a mouse, a dongle to connect to the PC, and a battery to make the mouse 
work, but in this article, the user with sound on the camera or on. Web camera and use hand gestures to control 
computer mouse functions. In the proposed method, the web camera captures and processes the captured images, 
then detects different hand gestures and gestures, and then performs the operation of the mouse. 


The main goal is to implement additional gestures so that in the future, users will be able to perform more 
tasks efficiently. This project proposes a system that only uses the appropriate hand to perform gesturesherefore, it 
will be possible in the future to improve the current system to use both hands to do different things. Many 
applications have improved significantly due to the rapid development of hand recognition systems. 


The AI mouse system is developed using the Python programming language, and OpenCV, which is a 
library for computer vision, is also used in the AI mouse system. In the proposed AI based mouse model, it uses the 
MediaPipe package for hand tracking and hand tracking, and also, the PyAutoGUI package is used to navigate the 
window screen. click, and scroll through the task. The results of the proposed model showed a very high level of 
accuracy, and the proposed model can work well in real applications using the CPU without using the GPU. 


2.RELATED WORK 


S. Vasanthagokul et al suggests a system that virtually control a computer or device through image 
processing using artificial intelligence (AI). The Proposed method focus on being able to virtually perform mouse 
cursor functions. One can track a finger's tip and the gesture it makes with the use of machine learning techniques 
[1]. Monali Shetty et al suggested an object tracking application, which makes use of the webcam to monitor the 
colored objects that help to interact with system. Computer vision libraries is used to identify an object, its 
movement and simulate those movements in real time tracking [2]. Sk. Jilani Basha et al presents a unique hand- 
gesture-based mouse control method. A more natural interaction between the user and the virtual mouse can be 
achieved using hand gestures. A collection of programmable rules is used by the suggested system to translate hand 
gestures into actions once hand tracking techniques have been used to record and track hand movements [3]. Rajat 
Dudhapachare et al suggested AI virtual mouse system solves the aforementioned problems by employing an 
external or digital camera to record hand motions. Voice assistants and object recognition are then used to improve 
the system's accuracy and detect hand points [4]. U Sairam et al suggests an application for a virtual mouse based 
that tracks various hand gestures, the user's hands are tracked by an inbuilt camera, which also recognizes specified 
gestures and triggers the associated mouse operations. PyAutoGUI and OpenCV have been used in the Python 
implementation of this system [7]. 


Li Wensheng proposed to implement virtual mouse through machine vision. We provide a set of messages 
that are created depending on the results of fingertip tracking, and these messages may be used to operate window- 
based apps. We employ an effective color-based method to track the movement of fingertips in real time. 
Additionally, we provide a framework that enables programmers to implement virtual mice in human-computer 
interaction [8]. Prachi Palsodkar et al proposed the efficiency of voice commands and hand gestures over traditional 
mouse and keyboard inputs for controlling mouse movement. The experimental results show that it is technically 
possible to use hand gestures and voice commands to control mouse movement. This method could be useful for 
various applications such as gaming, accessibility, virtual and augmented reality, industrial and manufacturing, and 
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healthcare [9]. S. Shriram et al proposes a computer vision-based artificial intelligence (AD) virtual mouse system 
that uses hand gestures and hand tip detection to do mouse function. 


The suggested system's primary goal is to replace traditional mouse devices with webcams or computers 
with built-in cameras to accomplish computer mouse cursor and scroll capabilities. The MediaPipe package is used 
by the AI virtual mouse system model to track the hands and their tips. The computer's window screen was navigated 
and operations like as left and right clicks and scrolling were performed using the Pynput, Autopy, and PyAutoGUI 
packages[10]. Vantukala VishnuTeja Reddy et al suggest, a solution for finger tracking in the real world is the 
primary goal, although computer cursor control is still done manually. Two techniques for tracking fingers are used 
in this study: colored fingertips and other by using bare fingers and hand gestures. This consists of three basic steps: 
tracking hand gestures, implementing an on-screen cursor, and detecting fingers using color identification [11]. The 
ability to manipulate the mouse pointer without the need for any additional hardware, such as gloves or a sensing 
device, and the capability to carry out all mouse functions, including scrolling, moving, selecting, left- and right- 
clicking. 


A webcam is used to record hand gestures for the Tensor Flow-based mouse control system, which uses an 
RGB color scheme. This approach enables users to move the mouse pointer on the screen, track the computer's 
webcam, and utilize various hand gestures to accomplish mouse functions like scrolling, dragging, and left and right 
clicks [5]. Roshnee Matlani et al suggest a method for manipulating the cursor's position without using any 
electronic equipment. While numerous hand motions will be used to perform tasks like clicking and dragging 
objects. Webcam is the only input device required for the proposed system. Together with additional tools, the 
proposed system will need to be used with OpenCV and Python [6]. 


3.ALGORITHMS USED 


A.KLT Tracking Algorithm 


An extraction method for features is the Kanade—Lucas—Tomasi (KLT) feature tracker. The KLT tracking algorithm 
is based on the concept of feature tracking, where distinctive features or points are identified in an initial frame, and 
then their locations are tracked across subsequent frames. These features may include parts, components, or other 
identifiable features that can be identified and tracked. It is suggested that the issue that has to be resolved is that 
conventional video recording solutions are frequently prohibitively costly. KLT finds the best match by using 
geographical dynamic information to direct the search. This technique analyzes minor matches between images and 
is faster than existing methods. 


B. K-Cosine Algorithm 


It presents a range-based corner detection method that helps to detect digital objects with large dimensions and 
different angles using curvature. The extent of curvature is represented by the measured K-cosine. Then, the parts 
of the article are marked. This is the K-Cosine Corner algorithm. Both of these algorithms are used in the proposed 
mouse model to achieve this goal. The KLT tracking algorithm is used to detect the user's hands and the K-cosine 
boundary tracking technique is employed to identify hand movements and locate landmarks. 


4.FLOW DIAGRAM AND ARCHITECTURE DIAGRAM 


The working model of proposed virtual mouse system is illustrated as a flowchart in figure 1. It includes 
steps viz. hand region detection, segmentation, fingertip detection and tracking and matching. Further the 
architecture diagram of proposed system is shown in figure 2, it lustrates overall implementation process of the 
proposed virtual mouse system. 
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Fig.1.Work Flow Diagram of proposed Gesture Controlled Virtual Mouse system 
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Fig.2. Architecture Diagram of proposed Gesture Controlled Virtual Mouse system 
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A. EXISTING SYSTEM 


The Existing system of the virtual mouse was successfully created and mouse operation were performed. 
These works even focused on converting the given input images into RGB image so that the system can detect the 
hand landmark points even easier and also it saves some processing time. This method even helps in detecting the 
hands a lot easier. The existing system uses packages like Opencv and autopy to make the system work. It uses 
Contour tracking algorithm. It has shown results and accuracy of 94%. It needs computer with higher specs, this 
system has a lot of lag in computer with lower specs. The existing system provides functionality with certain hand 
gestures and these hand gestures are all unique and they denote the different mouse operations. 


The Existing System of this virtual mouse is liable only with certain actions to be performed by the mouse 
like it can be used just to navigate, left and right click of the mouse actions, it cannot perform actions such as double- 
click, click and drag, scrolling. It uses KLT tracking algorithm to detect the hand and also it uses Contour tracking 
algorithm to detect the fingers and act towards it. This system was proposed by NamataraKade, Nisha. It uses 
packages like mediapipe, OpenCV and autopy to perform actions such as hand detection, finger detection and 
performing actions on the mouse using the given modules. It has a high accuracy of the mouse actions, but the 
response time of the virtual mouse is very slow. Also, system was not able to perform the mouse actions under less 
lighting conditions. 


Even though this system has a high reliability and accuracy of the mouse operations it still needs a lot to 
work on since the process and response time of the virtual mouse is slow and it cannot be used in the day-to-day 
basis, especially in the Augmented and Virtual Reality world. 


5.PROPOSED WORK 


The proposed method aims to provide a software-based virtual mouse that may be controlled virtually 
through hand gesture recognition. This proposed work mainly focus on the problems faced by the existing model. 
The proposed use algorithms KLT tracking and K-cosine algorithm. These algorithms, definitely reduce the process 
and response time of the virtual mouse. The proposed system must be able to provide the user with more mouse 
operations than the proposed system, it should have double click, scrolling, click and dragging, in order to put it in 
simple words unlike the existing system the proposed system should allow the user to perform all the mouse 
operations that can be done by an physical mouse on the virtual mouse. The proposed system has three modules 
they are (1) Hand Detection, (2) Hand Landmark Detection, (3) Gesture Recognition, (4) Virtual Mouse. 


Hand Detection 


The first and foremost step in the program is to detect the hand that is to be shown in front of a camera. 
This is achieved by using the OpenCV module, this module is used mostly for computer vision and this helps the 
computer to record or capture the hand in front of the camera. Once the camera starts capturing, the KLT tracking 
algorithm will detect the hand. After detection it is lock on to that hand and if there occur two hands detected by the 
system then, the hand which gets captured at first has high priority and that hand gets locked and the system only 
reads that hand even if there are more than one hand detected by the system. 


Hand- Land mark point Detection 


The Recorded hand is then using KLT Algorithm is landmarked. In this step each point in the hand is 
landmarked in order for the program to recognise the hand gesture. Each joint of the hand is a landmark and we try 
to mark these landmarks and display them on the screen using OpenCV. These landmark points are based on the 
position of the joint on the finger-tips using this we can easily recognise the hand gesture. 
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0. WRIST 11, MIDDLE FINGER_DIP 
1, THUMB_CMC 12, MIDDLE FINGERTIP 
2, THUMB_MCP 13. RING_FINGER MCP 
3, THUMB_IP 14, RING_FINGERPIP 

4, THUMB_TIP 15, RING_FINGER_DIP 


5, INDEX.FINGER.MCP 16, RING FINGERTIP 
6,INDEXFINGER PIP 17, PINKY. MCP 
T.INDEX.FINGERDIP 18, PINKY_PIP 
B.INDEXFINGERTIP 19, PINKY. DIP 
9, MIDDLEFINGER.MCP 20, PINKY_TIP 

10. MIDDLE_FINGER PIP 


Fig.3. Hand Landmark points 


C. Hand Gesture Recognition 


In this stage using the landmark points and also with the help of K-cosine Algorithm, the program 
understands the Hand Gesture. The purpose of the hand gesture is to make the program act accordingly to the 
gesture. The system understands the gesture with the help of the landmark points. For example, if the distance 
between the landmark points 8 and 12 are less than the system recognises the gesture as double click. The following 
are the gestures and actions provided to the Virtual Mouse through the program. 


These are the hand gesture and their action actions. All of these hand gestures are customizable. Each and every 
hand gesture is recognized by the landmark points in the hand. These gestures are understanded by the system 
through the hand landmark points with the help of K-cosine algorithm. 


Double-Click Right-Click 


Fig. 4. Various types of Hand Gesture Recognition 


D.Working of Virtual Mouse 


The proposed system act accordingly towards the hand gesture recognised through the k-cosine algorithm, 
in order to do this, the package or module named pyautogui Is used. This package assigns the actions to the virtual 
mouse. Pyautogui has access over both keyboard as well as mouse. While running the program simultaneous action 
of both proposed virtual mouse and the physical mouse can be accessed. 
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6 PERFORMANCE ANALYSIS. 


When compared with the existing system the proposed work has proved to be working more efficiently and 
less time consuming. As described earlier, the proposed work can work in both light and dim light settings. 


(b) 


Fig.5. Performance Analysis in (a) Bright Light and (b) Dim Light 
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It is also very well proven to work in long distances upto 20ft. It also works perfectly when there is more than one 
hand detected on the screen. Comparison of virtual mouse systems in the literature is still primitive. Since only a 
few literature and public databases are available, comparisons between methods are difficult. This system is offered 
on a laptop with a Ryzen3u processor with a clock cycle of 2.10 Ghz and 8 GB of RAM. This process was done in 
Python and the interpreter used was VScode. The tracking process was done at a speed of 30 frames per second. In 
this experiment, 10 subjects performed various quick movements and evaluated the results. The data is collected 
from different screen sizes and different computer models with different specifications and operating systems to 
show that the proposed model fits the application. There are four computer resolutions as follows: 1028 x 1024, 
1600 x 1200, 1900 x 1200 and 1680 x 1050. On each computer, 10 to 20 cases are recorded with different lights. 
Each of the gesture was done 10 times by 10 different individuals. All participants were right-handed for simplicity 
and accurate detection. The following table shows us the noted result of this experiment. It was noted that the Double 
click and Navigation has 100% accuracy. The Grab and drag action have the lowest accuracy of 88% and 
improvements are need to be done in this area. Other than that, left click and right click operations were done 
smoothly without any problem. The proposed work has an overall accuracy of 96.8%. When compared with other 
related works, proposed work has high accuracy level. 


Table 1: Testing with different mouse action 


HOME TIMES SUCCESS FAILURE ACCURACY 
PERFORMED 
Navigation 100 100 0 100% 
Left Click 100 95 5 95% 
Right Click 100 98 2 98% 
Double Click 100 100 0 100% 
Grab and drag 100 88 12 88% 
No movement 100 100 0 100% 


Table 2: Comparison between proposed work and existing works 


APPROACH ACCURACY 
Real-time virtual mouse system using RGB-D images 
2 ; : 95% 
and fingertip detection [11] 
A Vision based Application for Virtual Mouse 718% 
Interface Using Finger-Tip. [12] : 
An Interactive Computer System with Gesture-Based 
78% 
Mouse [13] 
Deep Learning-Based Real-Time AI Virtual Mouse 
System Using Computer Vision to Avoid COVID-19 94% 
Spread [9] 
Our Approach 96.8% 


With the help of the table 2 it is proven that accuracy level of proposed work is greater than that of the related works. 
This illustrates that the proposed work is the best work in the field of virtual mouse. 
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Fig.6. Accuracy comparison for different mouse actions 


Figure 6. Graph shows us the bar graph of the accuracy between the performance of the mouse on the 
conducted experiment. The graph clearly shows us that there is improvement needed in areas of Grab and Drag. 
Improvement is also needed in sensitivity of the virtual mouse. 


7.CONCLUSION 


The proposed system uses K-cosine border tracking and KLT algorithm to achieve the goal and by using 
this algorithm the success rate is raised upto 90%. The virtual mouse would be the future in mouse technology. As 
the AR and VR or Metaverse is growing rapidly and as it very imminent the virtual mouse would be a very 
innovative and easy and new to the mouse technology. In conclusion, the virtual mouse is a technology that has 
greatly transformed the way interact with computers and other digital devices. It serves as a convenient alternative 
to traditional physical mouse, offering users increased flexibility and accessibility in navigating and controlling 
their devices. The ability of a virtual mouse to eliminate a physical mouse is one of its advantage, this makes it 
especially helpful in scenarios where there is a lack of space or when a physical mouse is impractical. It enables 
users to operate their device with basic hand gestures or movements, often through the utilization of a built-in 
camera. 

However, despite these advantages, virtual mouse also has their limitations. They rely heavily on hardware 
capabilities, such as camera quality, which may vary across devices. Furthermore, they may not be suitable for 
certain tasks or applications that require high-speed and precise movements, such as gaming or graphic design. In 
conclusion, the virtual mouse has revolutionized computer interaction by providing a flexible, accessible, and 
customizable alternative to traditional physical mice. While it may not completely replace physical mice in all 
scenarios, it offers significant benefits in terms of convenience, adaptability, and accessibility, making it a valuable 
tool in the modern digital landscape. 
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